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METHOD AND APPARATUS FOR IMAGE METADATA ENTRY 

Field of the Invention 

The present invention relates generally to graphical processing and, in particular, to 
a method and apparatus for associating metadata with a plurality of digital images using a 
graphical user interface. The present invention also relates to a computer program 
product including a computer readable medium having recorded thereon a computer 
program for associating metadata with a plurality of digital images using a graphical user 
interface. 

Background 

Digital photography has become increasingly popular in recent times. One reason 
for the popularity of digital photography is that digital photographs do not require 
traditional development, with the associated cost and inconvenience. Such digital 
photographs can be produced and edited easily using readily available digital image 
software applications. Further, in contrast to traditional photographs, digital photographs 
are available for viewing and/or use almost immediately, upon the reading of an 
associated film diskette, by a personal computer (PC), or display device. 

As a result of the above, together with the ever-increasing use of digital images on 
the Internet, large databases of digital images are being assembled for both personal and 
commercial use. As with conventional photography, the need to annotate and catalogue 
the ever-increasing number of digital images is of paramount importance in order to allow 
ease of access and use. 

One method of facilitating the annotation of digital images is to generate 
"metadata" with the image. Metadata is information about the content of digital images 
or even video. For example, an image depicting a beach scene can include a short textual 
description such as "a picture of a beach", the name of a person in the image or a date and 
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time that the image was captured. Many Internet image search sites search on metadata 
content descriptions to locate digital images for display. 

Some digital cameras automatically generate metadata in the form of a date and 
time, which is generally included in the file name of a digital image when the image is 
5 stored and/or displayed (e.g. 12Nov_l.jpg). However, the automatically generated date 
and time says nothing about the content and/or event depicted by the digital image and 
therefore provides only limited assistance in annotating, cataloguing and searching for the 
digital image. 

Conventionally, a text entry method of generating metadata for digital images has 
10 been used to annotate large numbers of digital images. Such a method requires a person 
to sort through a database of digital images, using a computer, and store a short textual 
label with each image indicating a subject and/or an event depicted by the corresponding 
digital image. However, the above conventional method is very labour intensive and thus 
time consuming. As a result, the sorting and labelling of digital images is often neglected 
15 due to the time required to individually process voluminous images. The photographer 
therefore runs a risk of accumulating a growing number of images, many of which are not 
readily accessible because of the absence of a convenient method of labelling. 

In view of the above, efficient methods for classifying such large numbers of 
images is becoming increasingly essential. 
20 One known method for classifying digital images utilises a hierarchical structure 

similar to the hierarchical directory or folder structure used by the operating system of 
most conventional computers. Such a hierarchical structure is used for classifying digital 
images at a fundamental level by creating a tree of aptly named directories or folders and 
moving the images to the appropriate target destinations. However, such a process is 
25 repetitive and laborious, since the process typically involves viewing each image and then 
either copying or moving the respective image to the relevant directory or folder. 
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A further disadvantage of the above classification method is that directory names 
are necessarily brief and not very descriptive. In addition, there is a problem in cross- 
referencing images, which are classified into more than one category. For example, if an 
image is to be classified into more than one category, then multiple copies of the image 
must be made to each of a number of relevant folders or directories. 

The disadvantages of the above classification method have resulted in various other 
methods being proposed in order to make the process of classifying and storing digital 
images easier and more efficient. One such method stores collections of links to digital 
image files using metadata for classification purposes. Another method utilises a 
hierarchical structure for storing groups of digital image files. Still further, another 
known method labels digital images as the images are stored in a memory of a 
conventional computer system. 

The benefits of storing metadata within digital image files or associating such 
metadata externally from one or more particular image files, using a link to the image 
files, are well known. For example, a number of image search methods are known 
ranging from general search methods, methods which allow for the extraction of metadata 
from an image, and one known method which converts search results into particular 
formats preferred by a user. 

The above-mentioned search methods go some way to aiding digital camera users in 
classifying and maintaining large sets of digital images. However, the above methods are 
generally targeted at sophisticated users such as librarians and other database maintainers, 
rather than inexperienced or casual home computer users who wish to maintain large 
collections of personal digital images without a commitment to learning new software or 
operating paradigms. 

Another known method for classifying images, involves displaying a plurality of 
icons such that each icon is associated with a portion of metadata. An icon is 
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subsequently selected depending on at least one subject of an image and the metadata 
associated with the selected icon is stored as an association of the subject of the image. 
However, this method suffers from similar disadvantages to those discussed above in that 
the method is laborious and time consuming. Each of the images to be annotated has to be 
5 generated to full screen resolution in order to determine the subject of the image. Further, 
metadata icons have to be individually selected and dragged to such a full screen 
resolution view of the image to associate the metadata of the dragged icon with the image. 

Thus, a need clearly exists for an efficient and easy method of classifying and 
storing digital images. 
10 Summary 

It is an object of the present invention to substantially overcome, or at least 
ameliorate, one or more disadvantages of existing arrangements. 

According to one aspect of the present invention there is provided a method of 
classifying one or more images, said method comprising the steps of: 
15 selecting an iconic representation of at least one image displayed on a graphical 

user interface; 

moving said iconic representation to a target position within an area defined by said 
graphical user interface, according to a classification of said image; and 

determining an association between said at least one image and at least one 
20 predetermined metadata item representing said classification, in response to said iconic 
representation being positioned at said target position. 

According to another aspect of the present invention there is provided a method of 
classifying one or more images, said method comprising the steps of: 

selecting an iconic representation of at least one image, displayed on a graphical 
25 user interface; 
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moving said iconic representation to a target position within an area defined by said 
graphical user interface, according to a classification of said image; y : 

creating an association between said at least one image and at least .one metadata 
item, in response to said iconic representation being positioned at said target position; and 
5 generating an iconic representation of said at least one metadata item representing 

said classification. >..i, : .:. 

According to still another aspect of the present invention there is provided a method 
of searching for at least one image from a plurality^f images,Tsaid method comprising the 
steps of ^ 
10 selecting an iconic representation of at least one metadata item displayed on a 

graphical user interface; "vVi'k ■ >~v.<;-)\ 

determining an association between said at least one metadata item and said at least 
one image; and v : ,r - . 

generating an iconic representation of said at: least one image, said iconic 
15 representation of said at least one image being adapted for display on said graphical user 
interface. * 

According to still another aspect of the present invention there is provided a 
graphical user interface for representing classificationTelatipnships between one or more 
images and one or more metadata items, said graphical user interfece comprising: 
20 selection means for moving at least one iconic representation of at least one of said 

images displayed on said graphical user interface, to a target- position within an area 
defined by said graphical user interface, according to a classification of sakWmage; and 

at least one portion for displaying an iconic representation of a metadata item 
representing said classification, said metadata data item being generated and displayed in 
25 response to said at least one iconic representation being positioned at said target position. 
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According to still another aspect of the present invention there is provided a method 
of classifying one or more images, said method comprising the steps of: 

selecting an iconic representation of at least one image displayed on a graphical 
user interface; 

dragging said iconic representation to a target position within an area defined by 
said graphical user interface, according to a classification of said image; 
dropping said iconic representation at said target position 

generating an iconic representation of at least one metadata item, depending on an 
association created between said at least one image and said at least one metadata item 
representing said classification. 

According to another aspect of the present invention, there is provided an apparatus 
for implementing any one of the aforementioned methods. 

According to still another aspect of the present invention there is provided a 
computer program product including a computer readable medium having recorded 
thereon a computer program for implementing any one of the-methods described above. 

Other aspects of the invention are also disclosed. 

Brief Description of the Drawings 

Some aspects of the prior art and one or more embodiments of the present invention 
will now be described with reference to the drawings and appendices, in which: 

Fig. 1 shows a graphical user interface, in accordance with one arrangement; 

Fig. 2 shows an example of classifying a plurality of images, using the user 
interface of Fig. 1; >; 

Fig. 3 shows a further example of classifying a plurality of images; 

Fig. 4 shows a still further example of classifying a plurality of images; 

Fig. 5 shows an example of an iconic search on a plurality of images, using the user 
interface of Fig. t; 
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Fig. 6 shows a further example of an iconic search; 

Fig. 7 shows an example of a compound iconic search on a plurality of images, 
using the user interface of Fig. 1 ; 

Fig. 8(a) shows an example of converting a search result into a new collection, 
using the user interface of Fig. 1 ; 

Fig. 8(b) shows a further example of classifying a plurality of images; 

Fig. 8(c) shows a step in the example of Fig. 8(b); 

Fig. 8(d) shows a further step in the example of Fig. 8(b); 

Fig. 8(e) shows a hierarchical structure formed during the example of Fig. 8(b); 

Fig. 9 shows an example of an inverse search, using the user interface of Fig. 1; 

Fig. 10 shows a further example of an inverse search; 

Fig. 11 shows an example of adding region metadata to an image, using the user 
interface of Fig. 1; 

Fig. 12 is a flow diagram showing a method of classifying one or more images; 

Fig. 13 is a flow diagram showing a method of linking an icon in the Icons window 
of Fig. 1(a) with a selected drop target; 

Fig. 14 is a flow diagram showing a method of searching on a plurality of images; 

Fig. 15 is a flow diagram showing a further method of searching on a plurality of 
images; 

Fig. 16 is a flow diagram showing a method of associating a region of an image 
with one or more metadata items; 

Fig. 1 7 is a flow diagram showing a method of editing a metadata item; 

Fig. 1 8 is a schematic block diagram of a general-purpose computer upon which 
arrangements described can be practiced; 

Fig. 19 is a flow diagram showing a method of removing metadata-image 
associations from images; and 
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Fig. 20 is a flow diagram showing a further method of searching on a plurality of 
images. 

Detailed Description including Best Mode 

Where reference is made in any one or more of the accompanying drawings to steps 
5 and/or features, which have the same reference numerals, those steps and/or features have 
for the purposes of this description the same function(s) or operation(s), unless the 
contrary intention appears. 

It is to be noted that the discussions contained in the "Background" section and that 
above relating to prior art arrangements relate to discussions of documents or devices, 
10 which form public knowledge through their respective publication and/or use. Such 
should not be interpreted as a representation by the present inventor(s) or patent applicant 
that such documents or devices in any way form part of the common general knowledge 
in the art. 

A method 1200 of classifying one or more images is described below with 
15 particular reference to Fig. 12. A method of searching on a plurality of selected images is 
also described with particular reference to Fig. 14. The described methods are preferably 
practiced using a general-purpose computer system 1800, such as that shown in Fig. 18. 
In particular, the processes of Figs. 1 to 17, 19 and 20, described below may be 
implemented as software, such as an application program executing within the computer 
20 system 1800. In particular, the steps of the methods described herein are affected by 
instructions in the software that are carried out by the computer. The instructions may be 
formed as one or more code modules, each for performing one or more particular tasks. 
The software may also be divided into two separate parts, in which a first part performs 
the described methods and a second part manages a user interface between the first part 
25 and the user. The software may be stored in a computer readable medium, including the 
storage devices described below, for example. The software is loaded into the computer 
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from the computer readable medium, and then executed by the computer. A computer 
readable medium having such software or computer program recorded on it is a computer 
program product. The use of the computer program product in the computer preferably 
effects an advantageous apparatus for implementing the described processes. 
5 The computer system 1800 is formed by a computer module 1801, input devices 

such as a keyboard 1802 and mouse 1803, output devices including a printer 1815, a 
display device 1814 and loudspeakers 1817, A Modulator-Demodulator (Modem) 
transceiver device 1816 is used by the computer module 1801 for communicating to and 
from a communications network 1820, for example connectable via a telephone line 1821 
10 or other functional medium. The modem 1816 can be used to obtain access to the 
Internet, and other network systems, such as a Local Area Network (LAN) or a Wide 
Area Network (WAN), and may be incorporated into the computer module 1801 in some 
implementations. 

The computer module 1801 typically includes at least one processor unit 1805, and 
15 a memory unit 1806, for example formed from semiconductor random access memory 
(RAM) and read only memory (ROM). The module 1801 also includes a number of 
input/output (I/O) interfaces including an audio-video interface 1807 that couples to the 
video display 1814 and loudspeakers 1817, an I/O interface 1813 for the keyboard 1802 
and mouse 1803 and optionally a joystick (not illustrated), and an interface 1808 for the 
20 modem 1816 and printer 1815. In some implementations, the modem 1816 may be 
incorporated within the computer module 1801, for example within the interface 1808. A 
storage device 1809 is provided and typically includes a hard disk drive 1810 and a 
floppy disk drive 1811. A magnetic tape drive (not illustrated) may also be used. A CD- 
ROM drive 1812 is typically provided as a non- volatile source of data. The 
25 components 1805 to 1813 of the computer module 1801, typically communicate via an 
interconnected bus 1804 and in a manner, which results in a conventional mode of 



610883.doc 



- 10- 

operation of the computer system 1 800 known to those in the relevant art. Examples of 
computers on which the described arrangements can be practised include IBM-PC's and 
compatibles, Sun Sparcstations or alike computer systems evolved therefrom. 

Typically, the application program is resident on the hard disk drive 1810 and is 
read and controlled in its execution by the processor 1805. Intermediate storage of the 
program and any data fetched from the network 1820 may be accomplished using the 
semiconductor memory 1806, possibly in concert with the hard disk drive 1810. In some 
instances, the application program may be supplied to the user encoded on a CD-ROM or 
floppy disk and read via the corresponding drive 1812 or 181 1, or alternatively may be 
read by the user from the network 1820 via the modem device 1816. Still further, the 
software can also be loaded into the computer system 1 800 from other computer readable 
media. The term "computer readable medium" as used herein refers to any storage or 
transmission medium that participates in providing instructions and/or data to the 
computer system 1800 for execution and/or processing. Examples of storage media 
include floppy disks, magnetic tape, CD-ROM, a hard disk drive, a ROM or integrated 
circuit, a magneto-optical disk, or a computer readable card such as a PCMCIA card and 
the like, whether or not such devices are internal or external of the computer 
module 1801. Examples of transmission media include radio or infra-red transmission 
channels as well as a network connection to another computer or networked device, and 
the Internet or Intranets including e-mail transmissions and information recorded on 
Websites and the like. 

The methods described herein may alternatively be implemented in dedicated 
hardware such as one or more integrated circuits performing the functions or sub 
functions of the described methods. Such dedicated hardware may include graphic 
processors, digital signal processors, or one or more microprocessors and associated 
memories. 
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The described methods provide a user with an intuitive graphical user interface for 
classifying and searching on a plurality of digital images. Multiple simultaneous 
metadata associations and compound searches can also be performed, using the described 
methods. Such operations can be performed using simple user actions, which will be 
5 familiar to inexperienced or casual computer users who typically want to perform such 
operations on digital images without a commitment to learning new software or operating 
paradigms. 

Metadata is associated with digital images in the described methods by selecting 
iconic or thumbnail representations of the images and dragging the iconic or thumbnail 

10 representations to a destination point to either create a new association for a collection of 
images, hereinafter referred to as "a collection", or to associate a pre-existing metadata 
item with the images. Specific metadata information can be encoded within a digital 
image, for instance as information appended to the image header within the associated 
image file. Alternatively, the metadata information can be maintained in separate files 

15 stored in memory 1806, as metadata records containing metadata descriptions and 
references to the associated image files. Such metadata records can include fields 
describing attributes of a particular metadata item such as a label representing the 
metadata item, a reference to an icon to which the item is associated (i.e., a metadata-icon 
association), a reference to an image to which the item is associated (i.e., a metadata- 

20 image association) and the type of metadata item represented by the record. 

Some examples of metadata types that can be associated with an identified image 
can include one or more of the following types: 

(i) A data string; 

(ii) The name of a person; 

25 (iii) The address of a location; 

(iv) Date/Time; and 
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(v) Actual location. 

The described methods can be implemented to classify digital images locally on a 
particular computer such as the computer 1 800 or on a plurality of remote computers (not 
shown) connected to the network 1 820. The described methods can also be implemented 
as a specific application program or as one module in a governing application program. 

In addition to classifying digital images, the described methods allow intuitive 
searches on the images in a similar manner. A user can select an icon representing a 
metadata item of interest, and all digital images associated with the metadata item can be 
displayed to the user, on the display 1814, for example, as a collection of associated 
images. Such a collection can itself form a metadata association for a plurality of images. 

In one embodiment, compound searches can also be performed by selecting a 
plurality of iconic metadata representations, in which case the intersection of all digital 
images associated with all selected metadata items can be displayed to a user. 

Inverse searches can also be performed by selecting one or more digital images, in 
which case a union of all metadata items associated with any selected images can be 
highlighted to a user. 

The methods of classifying and searching on a plurality of digital images will be 
described in more detail below by way of example. 

Fig. 1 shows a graphical user interface 100 comprising two windows 101 and 103, 
which can be presented to a user on the display 1824, for example. The window 101 is 
titled "Icons" and has a client area 102, as known in the relevant art, which can be sized 
by a user in a conventional manner. Icons representing individual items of digital image 
metadata can be displayed within the client area 102 of the window 101 . 

As will be explained below, each of the icons displayed in the icons window 101 
has an image association list, which lists one or more images associated with a particular 
icon. The association list can be stored in memory 1806 and can be updated each time 
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one or more images are dropped onto an icon using the mouse 1803. Further, each icon 
displayed in the icons window can have one or more items of metadata associated with 
the icon. 

The items of metadata associated with the icons can be stored in a central database, 
5 for example, in memory 1806. Alternatively, a database can be situated remotely and 
accessed via the network 1820. Each metadata item in such a database can include a 
record, as described above, specifying a reference to an icon to which the particular 
metadata item is associated. 

The window 103 of the user interface 100 is preferably titled "Search Results" and 
10 also has a client area 104 of a size convenient to users. Thumbnail representations of 
images to be classified and images satisfying search criteria can be displayed in the 
window 103. Fig. 1 shows a number of thumbnail representations of unclassified images 
105, 106, 107, 108 and 109, which can be classified using the methods to be described. 

Fig. 12 is a flow diagram showing the method 1200 of classifying one or more 
15 images in accordance with one arrangement. The method 1200 is preferably implemented 
as software resident on the hard disk drive 1810 and being controlled in its execution by 
the processor 1805. The process begins at step 1201, where one or more thumbnail (or 
iconic) representations of images (i.e., image files) can be selected, dragged and dropped 
in either of the windows 101 or 102, using the mouse 1803. At the next step 1203, if the 
20 images are dropped within the client area 104 of the search results window 103, then the 
method 1200 proceeds to step 1204. Otherwise, the method 1200 proceeds to step 1206. 

At step 1204, the processor 1805 displays the thumbnail representations of the 
selected images within the window 103. Then at the next step 1205, the images dropped 
in the window 103 remain selected (i.e., highlighted as known in the relevant art), 
25 implying that further actions follow the selection of the images, as will be described in 
further detail below. The method 1200 concludes after step 1205. 



6l0883.doc 



. 14- 

At step 1206, if the processor 1805 determines that the images have not been 
dropped within the client area of the Search Results window 103 or the Icons window 
101, then the method 1200 concludes. Otherwise, if the selected images were dropped 
within the client area 102 of the Icons window 101, then the method 1200 proceeds to 
step 1208. At step 1208, if the images were dropped onto an icon already existing in the 
window 101, then the method proceeds to step 1209. Otherwise the method 1200 
proceeds to step 1211. 

At step 1209, references to the dropped images are added to an association list 
corresponding to the existing icon, and the method 1200 concludes. As a result, the 
dropped images are also associated with one or more items of metadata represented by the 
icon. The association between the dropped images and the metadata items (i.e., the 
metadata-image associations) can be implemented as a link (e.g. a pointer or reference) 
between the images and the metadata items, stored together with the particular metadata 
items in memory 1806, for example. 

As will be explained in detail below, metadata-image associations can be 
represented by a hierarchical tree structure 805, for example, as seen in Fig. 8(e). The 
structure 805 preferably comprises nodes (e.g. 806), where each node can contain: 

(i) Metadata information; and 

(ii) One or more sub nodes. 

Images and corresponding image files represented by thumbnail representations 
can be associated with nodes at the leaf (e.g. 807) of such a tree structure 805. Leaf nodes 
can also be associated with other file types such as audio and video files. Metadata items 
represented by icons (e.g. an icon 809) can be associated with each branch node (e.g. 
806). Thus, each branch of the hierarchical structure 805 contains metadata information 
that applies to a sub tree (not shown) below that branch. 
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Any image being a descendant of a branch is associated with the metadata item 
corresponding to the branch. A collection of metadata items can therefore be stored in 
memory 1 806 in a form representing a single tree structure. Such a collection can be 
stored in a central database locally within the computer 1800 or accessed over the 
5 network 1820. The tree structure 805 can be readily read to and from a file stored on the 
hard disk drive 1810 for persistence between operations. 

If the images selected at step 1201, are dropped onto an empty point within the 
client area 102 of the icons window 101, at step 1208, then the method 1200 proceeds to 
step 121 1. At step 121 1, the processor 1805 generates a new icon representing an item of 

10 metadata and the method 1200 concludes. The item of metadata represented by the icon 
generated at step 1211 can be read from the file header of one or more of the dropped 
images. Alternatively, the processor 1805 can read a reference, associated with the 
dropped images, to an item of metadata stored in memory 1806. At the next step 1212, a 
reference (i.e., metadata-image association ) to the item of metadata generated at step 

IS 121 1 is stored in memory 1806, and the method 1200 concludes. 

Continuing the example of Fig. 1, Fig. 2 shows three of the images 105, 106 and 
107, which have been selected and dragged to a point 204 within the client area 102 of the 
icons window 101. As a result, an icon 205 (i.e., labelled "iO") representing a metadata 
item is generated by the processor 1805, as at step 1211 of the method 1200. The 

20 metadata item represented by the icon 205 can be read from the file header of each of the 
images 105, 106 and 107. Alternatively, the processor 1805 can read a reference, 
associated with the dropped images 105, 106 and 107, to an item(s) of metadata stored in 
memory 1806. A collection has thus been generated, where the collection contains the 
selected images 105 to 107. The metadata item(s) associated with the selected images 

25 105 to 107 has not been initialised. The initialisation of metadata will be described 
below. 
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Multiple images can preferably be selected by pressing a key (e.g. the control key) 
on the keyboard 1802, while clicking the mouse 1803 on each thumbnail representations 
of the images in turn or sweeping the mouse 1803 over an area that contains the 
thumbnail representations representing the multiple images. 

To initialise the metadata item associated with the images 105 to 107, the user can 
double click on the icon 205 in a conventional manner or select the icon 205 and press a 
Properties Button, as known in the relevant art, to launch a Metadata Editor window (not 
shown). The Metadata Editor window (not shown) can be used to display the metadata 
fields (e.g. label, icon, type etc) of the metadata record associated with the icon 205 
selected. Such a Metadata Editor window can allow a suitable and readily identified 
thumbnail representation to be associated with the metadata item. The Metadata Editor 
window can also allow a user to select the type of metadata and the value of the metadata. 

Alternatively, a metadata item may be initialised by prompting a user to select an 
appropriate icon. Further, a default thumbnail icon can be generated and displayed in the 
icons window 101, when a new icon (e.g. the icon 205) and metadata item is being 
generated. The default icon can be replaced by an appropriate thumbnail representation at 
a later time through some convenient mechanism such as right clicking on the default 
icon. The label (e.g. 'i0') associated with an icon is preferably visible and editable as a 
text box. A selected image or an image selected first from any plurality of images can 
form a default thumbnail icon. Further, an abbreviation of such a selected image or the 
first selected image can make a suitable label for such a default icon. 

Continuing the example of Figs. 1 and 2, the classification of the images 107 and 
108 can be performed by selecting the images 107 and 108, dragging the images 107 and 
108 into the client area 102 of the window 101, and dropping the images 107 and 108 on 
the existing icon 205, as at steps 1202 to 1208 of the method 1200. As the image 107 is 
already associated with the icon 205 and the corresponding metadata item, no further 
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processing is performed on the image 107. Preferably, no error conditions are generated 
by the processor 1805 in this instance. However, in contrast to the image 107, the image 
108 is foreign to the set of images associated with the icon 205. Thus, the image 108 is 
added to the image association list of the icon 205 and a metadata-image association is 
added to the metadata item record corresponding to the icon 205. As such, the image 108 
is added to the collection of images associated with the icon 205. 

As seen in Fig. 4, the two images 106 and 109 can be selected and dragged in a 
conventional manner to an empty point 403 within the icons window 101 client area. As 
a result, another new metadata item is generated by the processor 1805, and an icon 404 
representing the metadata item (i.e., labelled "il") is generated. Another collection has 
thus been generated containing the selected images 106 and 109. This further collection 
is associated with the new item of metadata, although again, the metadata item does not 
have to be initialised at the time that the collection is generated. The metadata item 
associated with the icon 404 can be initialised as described above for the icon 205. 

Fig. 13 is a flow diagram showing a method 1300 of linking an icon (e.g. the icon 
205) in the Icons window 101 with a selected drop target (e.g. the icon 404). The method 
1300 is preferably implemented as software resident on the hard disk drive 1810 and is 
controlled in its execution by the processor 1805. The process begins at step 1302, where 
one or more icons (e.g. the icon 205) in the icons window 101 are selected, dragged and 
dropped, in a conventional manner using the mouse. 1803. At the next step 1303, if the 
icons are dropped within the client area 102 of the icons window 101, then the method 
1300 proceeds to step 1304. Otherwise, the method 1300 proceeds to step 1306. 

At step 1304, the processor 1805 deletes the dropped icons, and the method 1300 
concludes. 

The method 1300 continues at step 1306, where if the icons (e.g. the icon 205) were 
dropped onto an existing icon (e.g. the icon 404) in the window 101, then the method 
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proceeds to step 1308. Otherwise the method 1300 concludes. At Step 1308, any 
metadata items and images associated with the dropped icons are associated with the 
existing icon. Such associations are formed by updating the metadata records of the 
existing icon to include references images associated with the dropped icons. Any future 
images dropped on the existing icon will be associated with all of the metadata items of 
the existing icon and the metadata items of the dropped icons that were associated with 
the existing icon in step 1308. The method 1300 concludes after step 1308. 

Fig. 14 is a flow diagram showing a method 1400 of searching on a plurality of 
selected images, in accordance with one embodiment. The method 1400 is preferably 
implemented as software resident on the hard disk drive 1810 and being controlled in its 
execution by the processor 1805. The process begins at step 1402, where one or more 
images (or thumbnail representations) are selected using the mouse 1803 in a 
conventional manner. As described above, multiple images can preferably be selected by 
pressing a key (e.g. the control key) on the keyboard 1802, while clicking the mouse 1803 
on each thumbnail image representation in turn or sweeping the mouse 1803 over an area 
that contains the thumbnails representing the multiple images. 

At the next step 1403 of the method 1400, if the selection of images occurs outside 
the search results window 103, then no further processing is executed and the method 
1400 concludes. Otherwise, if the selection of images occurs within the client area 104 of 
the search results window 103 then the method 1400 proceeds to step 1405. 

At step 1405, the processor 1805 determines the union of all metadata items 
associated with any of the selected images. Then at the next step 1406, any icons 
associated with those metadata items of the selected images are highlighted, in a 
conventional manner, in the icons window 101. 

The method 1400 is an example of an inverse search. For example, turning now to 
Fig. 9 an image 106 selected in the search results window 103 (i.e., the thumbnail 
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representation of the image 106 is highlighted in a conventional manner (e.g. shading)). 
Further, all metadata icons (e.g. the icons 205, 404 and 901) associated with the selected 
image 106 are themselves highlighted. In other words, selecting one or more images in 
the search results window 103 results in the highlighting of all metadata icons associated 
with those images. Inverse searching in this manner allows a user to quickly and easily 
determine, which items of metadata are associated with a particular image or set of 
images in a visual manner. 

An image need not be displayed in the search results window 103 to perform an 
inverse search. For example, Fig. 10 shows the image 107 dragged (i.e., as indicated by 
the arrow 1001) from outside the windows 101 and 103 and dropped within the client area 
104 of the search results window 103. As a result, the image 107 is selected and 
highlighted in accordance with the method 1400. Therefore, an inverse search can be 
performed by the selection of an image 107, which indicates that the metadata item 
represented by icon 205 is associated with the image 107. Alternatively, the user can 
choose to search for the intersection of metadata items associated with the selected 
images, when performing an inverse search. 

As described above, the association of metadata items with images forms a 
symmetrical relationship. That is, associating an image with a metadata item represented 
by an icon, allows a user to classify the images. Further, listing those images associated 
with a set of metadata items and/or listing those metadata items associated with a set of 
images, allows a user to search on a plurality of digital images. 

Fig. 15 is a flow diagram showing a further method 1500 of searching on a plurality 
of images. The method 1500 is preferably implemented as software resident on the hard 
disk drive 1810 and being controlled in its execution by the processor 1805. The process 
begins at step 1502, where one or more icons are selected using the mouse 1803, in a 
conventional manner. Multiple icons can preferably be selected by pressing a key (e.g. 
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the control key) on the keyboard 1802, while clicking the mouse 1803 on each icon in 
turn or sweeping the mouse 1803 over an area that contains the icons. At the next step 
1503 of the method 1400, the processor 1805 determines the intersection of all images 
associated with any of the selected icons. The images can be determined at step 1503 by 
reading image references out of the association lists of each of the selected icons. Then at 
the next step 1504, thumbnail representations of those images determined at step 1503, 
are displayed in the search results window 103, and the method 1500 concludes. A new 
collection based on the images determined at step 1503 (i.e., the search results), can be 
created in the manner described above. 

The method of 1 500 is an example of a simple forward search, in accordance with 
one embodiment. For example, Fig. 5 shows the icon 205 selected and highlighted in a 
conventional manner (i.e., by shading), as at step 1502 of the method 1500. Selecting the 
icon 205 results in the images 105, 106, 107 and 108 associated with the icon 205, being 
displayed in the Search Results window 103. As described above with reference to Figs. 
2 and 3, the images 105, 106, 107 and 108 were previously classified as belonging to the 
icon 205 and the metadata items of the icon 205. 

Fig. 6 shows an example of another simple forward search performed by a user 
selecting the icon 404. As a result of the selection, the images 106 and 109 previously 
classified as belonging to the icon 404 are displayed in the search results window 103. In 
this instance, the search results window 103 is preferably cleared (i.e., removing previous 
search results) before displaying the current search results (i.e., the images 106 and 109). 

Fig. 7 shows an example of a compound forward search. A compound forward 
search is executed by the processor 1805 if more than one icon (e.g. both of the icons 205 
and 404) are selected. In this instance, thumbnail representations of each image 
associated with each of the icons 205, 404 representing metadata items, are displayed in 
the Search Results window 103. In the present example of Fig. 7, the image 106, which is 
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common to both icons 205 and 404, is displayed in the window 103. As such the result 
of the compound search is defined as the intersection of the association lists, 
corresponding to the selected icons, with all selected metadata items. The selection of one 
or more metadata icons, as described above, allows a user to perform compound searches 
quickly and intuitively. 

Alternatively, a user can choose to search for the union of association lists 
associated with metadata items. Multiple images (e.g. the image 106) from the search 
results window 103 can be classified simultaneously by selecting such images in the 
window 103 before dragging the selected images into the window 101. For example, Fig. 
8(a) shows the image 106 being dragged from the search results window 103 onto an 
empty point 802 within the icons window 101. As a result, a new uninitialised metadata 
item represented by icon 803 and associated with the image 106, is generated by the 
processor 1805. The new metadata item can be initialised as described above. 

Similarly, one or more images can be dragged from the search results window 103 
onto an existing icon (e.g., the icon 205) to associate those dragged images with the 
particular metadata item(s) represented by the icon. 

As described above, one or more images can be associated with one or more 
metadata items (i.e., classified) using the mouse 1803 in a conventional drag and drop 
manner. The images can be selected and dragged from within the window 103. 
Alternatively, thumbnail representations of images can be selected from outside the 
graphical user interface 100. For example, images can be selected from another 
application being executed on the computer 1800 or on a remote processor accessed via 
the network 1820. 

Icons (e.g. the icons 205, 404 and 803) can be deleted by dragging the icons outside 
the icons window 101 and dropping the icons, using the mouse 1803. Alternatively, icons 
can be deleted using some other user action such as right clicking the mouse 1803 on the 
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icons to be deleted to bring up a context menu, as known in the relevant art, and selecting 
a "delete icon" option. 

Icons that are selected, dragged and dropped on top of another existing icon are 
associated with the existing icon and the metadata items represented by the existing icon. 
5 For example, if the icon 205 is dragged and dropped onto the icon 803, then the icon 205 
is associated with icon 803. In this case, icon 205 is termed the "child icon" and icon 803 
is termed the "parent icon". As a result, any further operations on metadata items 
associated with the icon 803 are associated with any images listed in the association list 
corresponding to the icon 205. However, the relationship between the icons 205 and 803 
10 is not commutative, in this instance. 

Dragging and dropping icons onto existing icons, as described above, creates a 
parent-child relationship between the icons. This relationship can be represented by a 
metadata tree structure (e.g. the structure 805 as seen in Fig. 8(e)). For example, the 
image 105 of an "A" and the image 109 of an "E" as seen in Fig. 8(b), can be dragged 
15 and dropped onto an empty point 807 in the Icons window 101. As a result, the processor 
1805 generates an uninitialised metadata item, represented by an icon 809, associated 
with the two images 105 and 109. In the present example, the new icon 809 is labelled 
"vwls" by the user, using a text box generated within the icon 809, for example. The icon 
809 can be used to describe a subset of vowels (i.e., "A" and "E"), in the present example. 
20 Continuing the present example, the user then selects, drags and drops the image 

106 of a "B", the image 107 of a and the image 108 of a"D'\ onto an empty point 81 1 
within the Icons window 811, as shown in Fig. 8(c). As a result a new icon 813 
representing an uninitialised metadata item is generated by the processor 1805, as seen in 
Fig. 8(d). The new icon 813 is subsequently labelled "cons" by the user to describe a 
25 subset of consonants (i.e., the images 106, 107 and 108, representing the letters "B", "C" 
and "D"). 
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Continuing the present example, the user selects the icons 809 and 813, drags and 
drops the icons 809, 813 (i.e., labelled "vwls" and "cons") onto an empty point 815 within 
the icons window 101, as shown in Fig. 8(d). As a result, a new icon 817 representing a 
new metadata item is generated and displayed in the window 101. The information fields 
(e.g. label, icon, type etc) of the new metadata item represented by the icon 817 are not 
yet initialised. However, these information fields can be initialised by the user on the 
basis that the icons 809 and 813 (i.e., "vwls" and "cons") are children of the icon 817. 
The icon 817 is labelled "letters" by the user, as seen in Fig. 8(e), and represents a subset 
of letters of the alphabet. The subset of characters represented by the icon 817 has been 
further specialised into subsets representing vowels and consonants. 

As described above, the relationship between the icons 809, 813 and 817 can be 
represented by the hierarchical tree structure 805. However, the relationship between the 
icons 809, 813 and 817 can be represented in any suitable form (e.g., a table). Further 
icons (not shown) can be similarly dragged and dropped onto the existing icons 809, 813 
and 817 to create further parent-child relationships between the further icons and the 
existing icons 809, 813 and 817. As such, a new uninitialised parent icon does not need 
to be created for these further icons. 

Double clicking on an image in the search results window 103 or selecting an image 
in the search results window 103 and pressing a 'Properties Button', can be performed by 
a user in order to generate an image view window 1100, as shown in Fig. 11. The 
window 1 100 can be titled "Image View" 1 101. The window 1 100 contains a client area 
1102 which shows a screen resolution representation 1103 of the letter "A", which was 
previously represented by the thumbnail representation 105, as described above. 

In one example, if a user drags the mouse 1803 in a path 1 104 that approximates the 
outline shape of the representation 1 103 (i.e. the shape of the letter "A"), and then selects 
one or more icons (e.g., 205, 404 or 901) within the Icons window 101, then the region 
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1105 within the path 1104 is associated with the one or more selected icons and 
corresponding metadata items. The region 1 105 is closed by the processor 1805 to form a 
closed outline described by spline curves. If the representation 1 103 was not previously 
associated with any of the corresponding metadata items then new metadata-image 
5 associations are created, by adding a reference to the image represented by the region 
1 105 to the association lists and metadata records of the selected icons. 

A person skilled in the relevant art would appreciate that any suitable method, for 
describing a region within an image (e.g. the region 1105) can be used. For example, a 
user can drag a rectangular outline or an outline of any other geometric shape, or single 
10 click region detection using the mouse 1803. Once the association with such a region 
has been created, then a modified form of inverse search can be performed from the 
image view window 1 100. In order to perform such an inverse search, a user can click on 
a pixel within the image including the created region (e.g. the region 1105), using the 
mouse 1803. As a result, the following icons will be highlighted in the Icons window 
15 101: 

(a) Those icons corresponding to metadata items associated with the region (e.g. the 
region 1 105) within which the user has clicked; and 

(b) Those icons corresponding to metadata items associated with the image, which 
includes the region but with no specific region metadata-image associations. 

20 Fig. 16 is a flow diagram showing a method 1600 of associating a region with one 

or more metadata items. The method 1600 is preferably implemented as software 
resident on the hard disk drive 1810 and being controlled in its execution by the processor 
1805. The process begins at step 1601, where an image (e.g. the image 1 103) within the 
search results window 103 is selected by double clicking on the image using the mouse 

25 1803. Alternatively, the image can be selected using a "Properties Button" or "menu 
item", as known in the relevant art. 
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At the next step 1603, an image view window (e.g. the window 1 100) is launched 
by the processor 1805 to show the image at screen resolution. Depending on the size of 
the image, the window 1 100 can include a scroll bar. The method 1600 continues at the 
next step 1604, where if a mouse pointer associated with the mouse 1803 is not dragged 
5 within the window 1 100 to define a region (e.g., the region 1 105), then the method 1600 
concludes. 

If a region (i.e., typically following an outline shape within the image) is defined 
within the window 1 100, then the method 1600 proceeds to step 1606. At step 1606, if an 
icon (e.g. the icon 205) is selected within the icons window 101, then the method 1600 

10 proceeds to step 1608. Otherwise, the method 1600 concludes. At step 1608, the region 
defined within the window 1100 at step 1604 is associated with the icon selected at step 
1606, in the manner described above, and the method 1600 concludes. 

During the execution of the method 1600, one or more icons can be selected 
without a search being performed and without updating the contents of the search results 

15 window 103. The method 1600 and any search are performed in two clearly defined and 
mutually exclusive states (i.e., when the Image View window 1 100 is either open or 
closed). 

Fig. 17 is a flow diagram showing a method 1700 of editing a metadata item. The 
method 1700 is preferably implemented as software resident on the hard disk drive 1810 
20 and being controlled in its execution by the processor 1805. The process begins at step 
1701, where an icon (e.g. the icon 205) within the icons window 101 is selected by double 
clicking on the icon using the mouse 1 803. Alternatively, the icon can be selected using a 
properties button or menu item, as known in the relevant art. 

At the next step 1703, a Metadata Editor window (not shown) is launched by the 
25 processor 1805 to display the metadata fields (e.g. label, icon, type etc) of the metadata 
record associated with the icon selected at step 1803. The method 1700 concludes at the 
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next step 1704 where the metadata fields are edited by a user and the metadata editor 
window is closed in a conventional manner using the mouse 1 803. 

Fig. 19 is a flow diagram showing a method 1900 of removing metadata-image 
associations from images. The method 1900 is preferably implemented as software 

5 resident on the hard disk drive 1 8 1 0 and being controlled in its execution by the processor 
1805. The process begins at step 1902, where one or more icons (e.g. the icon 205) 
within the icons window 101 are selected by double clicking on the icons using the mouse 
1803. In response to the selection of the icons, the processor 1805 determines the 
intersection of all images associated with any of the selected icons, in accordance with the 

10 method 1500. Also at step 1902, those images determined to be associated with the 
selected icons, are displayed in the search results window 103, as thumbnail 
representations (e.g., the thumbnail representations 105 to 109). Then at the next step 
1903, one or more of the thumbnail representations displayed at step 1902, are selected. 
The thumbnail representations can be selected by right clicking the mouse 1803, for 

15 example, to bring up a context menu. A "remove associations" option can be selected 

from such a context menu. 

The method 1900 continues at the next step 1904, where the metadata-image 
associations previously stored in memory 1806 corresponding to the images represented 
by the displayed thumbnail representations and each of the metadata items represented by 

20 the selected icons, are removed from the metadata database stored in memory 1806, for 
example. The method 1900 concludes at the next step 1905, where the thumbnail 
representations displayed in the search results window 103, are refreshed with a new 
search to visually confirm the new state of the metadata database to the user. That is, any 
thumbnails representing images, which were removed from the metadata database, are 

25 removed from the search results window 103. 
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Alternative methods of removing metadata-image associations can be used. For 
example, a set of icons can be selected and images determined to be associated with the 
selected icons, can be displayed in the search results window 103, as thumbnail 
representations (e.g., the thumbnail representations 105 to 109), in accordance with the 
5 method 1500. The displayed thumbnail representations can then be selected and dragged 
from the search results window 103 and dropped outside the window 103. As a result the 
images represented by the selected thumbnails can be removed from the association lists 
corresponding to the selected icons. 

Fig. 20 is a flow diagram showing a further method 2000 of forward searching on a 
10 plurality of images. The method 2000 is preferably implemented as software resident on 
the hard disk drive 1810 and being controlled in its execution by the processor 1805. The 
process begins at step 2002, where search settings can be modified. Such settings can 
include instructions for handling specific search criteria (e.g. whether the search is to 
contain the union or intersection of target images). Also at step 2002, an icon selection list 
15 is configured within memory 1806 and is initialised to empty. Then at the next step 2005, 
if one or more icons in the icons window 101 are selected, the method 2000 proceeds to 
step 2005. Otherwise the method 2000 concludes. 

At step 2005, if the processor 1805 determines that a shift key of the keyboard 1802 
was depressed when the one or more icons were selected at step 2003, then the method 
20 2000 proceeds to step 2008. Otherwise the method 2000 proceeds to step 2006, where the 
processor 1 805 re-initialises the icon selection list to only contain the icon selected at step 
2003. A person skilled in the relevant art will appreciate that any other suitable key (e.g. 
the control key) can be used to perform the test at step 2005. 

The method continues at step 2008, where a reference to the selected icon(s) is 
25 added to the icon selection list. Then at the next step 2007, the processor 1805 
determines the intersection of all images associated with any of the selected icons, in 
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accordance with the method 1500. At the next step 2009, those images determined to be 
associated with the selected icons, are displayed in the search results window 103, as 
thumbnail representations (e.g., the thumbnail representations 105 to 109), and the 
method 2000 returns to step 2003 to await further icon selections. 
5 The aforementioned preferred method(s) comprise a particular control flow. There 

are many other variants of the preferred method(s), which use different control flows 
without departing the spirit or scope of the invention. Furthermore one or more of the 
steps of the preferred method(s) may be performed in parallel rather sequential. 

The foregoing describes only some embodiments of the present invention, and 
10 modifications and/or changes can be made thereto without departing from the scope and 
spirit of the invention, the embodiments being illustrative and not restrictive. For 
example, the methods described above can also be implemented as an interface embedded 
within an existing application or as a standalone application. Such applications can be 
executed either on an individual computer (e.g. the computer 1800) or on a number of 
IS computers (not shown) across a network (e.g. the network 1820). 

In the context of this specification, the word "comprising" means "including 
principally but not necessarily solely" or "having" or "including", and not "consisting 
only of. Variations of the word "comprising", such as "comprise" and "comprises" have 
correspondingly varied meanings. 
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The claims defining the invention are as follows: 

1. A method of classifying one or more images, said method comprising the steps of: 
selecting an iconic representation of at least one image displayed on a graphical 
5 user interface; 

moving said iconic representation to a target position within an area defined by said 
graphical user interface, according to a classification of said image; and 

determining an association between said at least one image and at least one 
predetermined metadata item representing said classification, in response to said iconic 
10 representation being positioned at said target position. 

2. A method according to claim 1, further comprising the steps of: 
generating an iconic representation of said metadata item; and 
displaying said metadata representation on said graphical user interface. 

3. A method according to claim 2, further comprising the steps of: 
selecting at least one further iconic representation of at least one further image 

displayed on said graphical user interface; 

moving said iconic representation to a position defined by said displayed metadata 
representation; and 

creating an association between said further image and said at least one metadata 

item. 

4. A method according to claim 1, further comprising the step of storing said 
25 association between said at least one image and said at least one metadata item. 



15 
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5. A method according to claim 4, wherein said association is stored locally to said 
graphical user interface. 



6. A method according to claim 4, wherein said association is stored remotely from 
5 said graphical user interface. 

7. A method according to any one of claims 1 to 6, wherein said iconic representations 
of said images are thumbnail images. 

10 8. A method of classifying one or more images, said method comprising the steps of: 
selecting an iconic representation of at least one image, displayed on a graphical 
user interface; 

moving said iconic representation to a target position within an area defined by said 
graphical user interface, according to a classification of said image; 
15 creating an association between said at least one image and at least one metadata 

item, in response to said iconic representation being positioned at said target position; and 

generating an iconic representation of said at least one metadata item representing 
said classification. 

20 9. A method according to claim 8, further comprising the step of displaying said 
metadata representation on said graphical user interface. 

10. A method according to any one of claims 8 or 9, further comprising the steps of: 

selecting at least one further iconic representation of at least one further image, 
25 displayed on said graphical user interface; 
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moving said iconic representation to a position defined by said displayed metadata 
representation; and 

creating an association between said further image and said at least one metadata 

item. 

5 

11. A method of searching for at least one image from a plurality of images, said 
method comprising the steps of: 

selecting an iconic representation of at least one metadata item displayed on a 
graphical user interface; 

10 determining an association between said at least one metadata item and said at least 

one image; and 

generating an iconic representation of said at least one image, said iconic 
representation of said at least one image being adapted for display on said graphical user 
interface. 

15 

12. A method according to claim 11, further comprising the step of displaying said 
iconic representation of said at least one image on said graphical user interface. 

13. A method according to claim 1 1, further comprising the steps of: 

20 selecting at least one further iconic representation of at least one further metadata 

item displayed on said graphical user interface; 

determining an association between said at least one further metadata item and at 
least one further image; and 

generating an iconic representation of said at least one further image for display on 
25 said graphical user interface. 
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14. A method according to claim 11, wherein said at least one metadata item is stored 
locally to said graphical user interface. 

15. A method according to claim 1 1, wherein said at least one metadata item is stored 
5 remotely from said graphical user interface. 

16. A method according to any one of claims 11 to 15, wherein said iconic 
representation of said image is a thumbnail image. 

0 17. A graphical user interface for representing classification relationships between one 
or more images and one or more metadata items, said graphical user interface comprising: 
selection means for moving at least one iconic representation of at least one of said 
images displayed on said graphical user interface, to a target position within an area 
defined by said graphical user interface, according to a classification of said image; and 

> at least one portion for displaying an iconic representation of a metadata item 

representing said classification, said metadata data item being generated and displayed in 
response to said at least one iconic representation being positioned at said target position. 

18. A graphical user interface according to claim 17, further comprising; 

a further selection means for selecting said iconic representation of said at least one 
metadata item displayed on a graphical user interface; and 

at least one further portion for displaying at least said iconic representation of said 
at least one image in response to said selection of said iconic representation of said at 
least one metadata item. 
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19. A graphical user interface' according to claim 18, wherein said further portion 
displays any further iconic representations of said one or more images, said further iconic 
representations being generated depending on determined associations between said one 
or more images and any other metadata items represented in said at least one portion. 

5 

20. A method of classifying one or more images, said method comprising the steps of: 
selecting an iconic representation of at least one image displayed on a graphical 

user interface; 

dragging said iconic representation to a target position within an area defined by 
10 said graphical user interface, according to a classification of said image; 
dropping said iconic representation at said target position 

generating an iconic representation of at least one metadata item, depending on an 
association created between said at least one image and said at least one metadata item 
representing said classification. 

15 

21 . Apparatus for performing the steps of any one of claims 1 to 20. 

22. A computer program comprising code for executing the steps of any one of claims 1 
to 20. 

20 

23. A method of classifying one or more images, said method being substantially as 
herein before described with reference to any one of the embodiments as that embodiment 
is illustrated in the accompanying drawings. 
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24. An apparatus for classifying one or more images, said apparatus substantially as 
herein before described with reference to any one of the embodiments as that embodiment 
is illustrated in the accompanying drawings. 

5 25. A graphical user interface for representing classification relationships between one 
or more images and one or more metadata items, said graphical user interface being 
substantially as herein before described with reference to any one of the embodiments as 
that embodiment is illustrated in the accompanying drawings. 

10 26. A computer program for classifying one or more images, said program being 
substantially as herein before described with reference to any one of the embodiments as 
that embodiment is illustrated in the accompanying drawings. 

DATED this Thirteenth Day of November 2002 
15 CANON KABUSHIKI KAISHA 

Patent Attorneys for the Applicant 
SPRUSON&FERGUSON 
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